Image and Collateral Text in Support of Auto-Annotation and Sentiment Analysis

We present a brief overview of the way in which image analysis, coupled with associated collateral text, is being used for auto-annotation and sentiment analysis. In particular, we describe our approach to auto-annotation using the graphtheoretic dominant set clustering algorithm and the annotation of images with sentiment scores from SentiWordNet. Preliminary results are given for both, and our planned work aims to explore synergies between the two approaches. 1 Automatic annotation of images using graph-theoretic clustering Recently, graph-theoretic approaches have become popular in the computer vision field. There exist different graph-theoretic clustering algorithms such as minimum cut, spectral clustering, dominant set clustering. Among all these algorithms, the Dominant Set Clustering (DSC) is a promising graph-theoretic approach based on the notion of a dominant set that has been proposed for different applications, such as image segmentation (Pavan and Pelillo, 2003), video summarization (Besiris et al., 2009), etc. Here we describe the application of DSC to image annotation. 1.1 Dominant Set Clustering The definition of Dominant Set (DS) was introduced in (Pavan and Pelillo, 2003). Let us consider a set of data samples that have to be clustered. These samples can be represented as an undirected edge-weighted (similarity) graph with no self-loops G = (V,E,w), where V = 1, . . . , n is the vertex set, E ⊆ V × V is the edge set, and w : E → R+ is the (positive) weight function. Vertices in G represent the data points, whereas edges represent neighborhood relationships, and finally edge-weights reflect similarity between pairs of linked vertices. An n × n symmetric matrix A = (aij), called affinity (or similarity) matrix, can be used to represent the graph G, where aij = w(i, j) if (i, j) ∈ E, and aij = 0 if i = j. To define formally a Dominant Set, other parameters have to be introduced. Let S be a nonempty subset of vertices, with S ⊆ V , and i ∈ S. The (average) weighted degree of i relative to S is defined as: